Spatially-Aware Displays For Computer-Assisted Interventions

ABSTRACT

Described herein are systems, methods, and techniques for spatially-aware displays for computer-assisted interventions. A Fixed View Frustum technique renders computer images on the display using a perspective based on a virtual camera having a field-of-view facing the display and automatically updates the virtual position of the virtual camera in response to adjusting the pose of the display. A Dynamic Mirror View Frustum technique renders computer images on the display using a perspective based on a field-of-view of a virtual camera that has a virtual position behind the display device. The virtual position of the virtual camera is dynamically updated in response to movement of a user&#39;s viewpoint located in front of the display device. Slice visualization techniques are also described herein for use with the Fixed View Frustum and Dynamic Mirror View Frustum techniques.

CROSS-REFERENCE TO RELATED APPLICATIONS

The subject United States Non-Provisional Patent application claimspriority to U.S. Provisional Patent Application No. 63/036,559, filedJun. 9, 2020, the entire contents of which are hereby incorporated byreference.

BACKGROUND

In the very early days of X-ray fluoroscopy, radiologists usedfluorescent handheld screens which were placed into the beam emanatingfrom the X-ray source passing through the patient. The image had adirect correlation to the X-ray source, the patient, the screen and theobserver, as it was observed directly where it was produced. Withadvances in imaging and display technology, it became possible to viewstatic or live images of the patient anatomy at any location, where amonitor could be positioned. This had the advantage of more practicalpatient and display positioning and reduced dose to theinterventionalist, but the intuitive perception of the spatialconfiguration between X-ray source, patient, screen, and the viewer waslost.

CT or MRI scans, which are inherently 3D, are often visualized as a setof three orthogonal slices along the anatomical axes of the patient oralong the axis of an instrument. Only recently have 3D images beenrendered on displays in operating rooms, often in addition to the slicevisualizations. These are often volume renderings from viewpoints, whichcan be defined and controlled by the user. Many surgeons prefer 2Dimages rather than 3D graphic renderings as they have been well trainedin interpreting them and thus this defines the current clinicalstandard.

Navigation in surgery has been introduced since the early 1990s andtakes advantage of different visualization techniques. Many systems showslices of pre- or intra-operative data with additional annotations.These annotations relate to a surgical plan, which needs to be carriedout in surgery. Such annotations are often an attempt to visualize deepseated anatomical targets and safe paths for the surgeon to reach thembased on pre-operative images. In such systems, when rendering images,the position of the surgeon and location of displays are not taken intoconsideration. The position of the display therefore does not affect thevisualization nor does the position of observer.

A series of related work is often referred to as Augmented Realitywindows (AR-windows). Prior techniques tracked semi-transparent displayfor medical in-situ augmentation, which incorporated a head tracker andstereo glasses. Such semi-transparent displays included a half-silveredglass pane, which reflected the image from a computer display. Othershave addressed the same problem of creating an AR-window on patientanatomy using a semi-transparent display between the patient and thesurgeon. However, such prior techniques replaced the half-silvered glasspane with an active matrix LCD. Later attempts approached the problemagain with a half-silvered glass pane, which rejected the projection oftwo DLP projectors generating high contrast images. Other systemscomprised of a tracked mobile opaque screen in which the position of thescreen affected the visualization. Once again, such prior systems wereplaced between the surgeon and the patient showing a slice view of theanatomy. However, in this system, the user's perspective is notconsidered, i.e. the image on the screen is merely two-dimensional,independent from the viewpoint of the surgeon.

Others have presented an AR visualization inspired by the dentists'approach for examining the patient's mouth without changing theirviewpoints. Such techniques identified that in some AR applications,rotating the object or moving around it is impossible. It was suggestedto generate additional virtual mirroring views to offer secondaryperspectives on virtual objects within an AR view. Spatially trackedjoysticks were utilized to move a virtual mirror which reflected thevirtual data like a real mirror within the AR view of a head mounteddevice (HMD).

SUMMARY

This Summary introduces a selection of concepts in a simplified formthat are further described below in the Detailed Description below. ThisSummary is not intended to limit the scope of the claimed subject matterand does not necessarily identify each and every key or essentialfeature of the claimed subject matter.

In a first aspect, a system for aiding in interaction with a physicalobject is provided, the system comprising: a display device defining aplane and being located on a first side of the physical object; and anavigation system coupled to a control system and being configured to:register computer images to the physical object; track a pose of thephysical object and the display device in a common coordinate system;and control the display device to render the registered computer imagesaccording to the tracked pose of the physical object and the displaydevice; and wherein a perspective of the rendering is based on a virtualcamera that has a virtual position located on a second side of thephysical object opposite to the first side, and wherein the virtualcamera has a field-of-view that faces the plane of display device, andwherein the virtual position of the virtual camera updates automaticallyin response to adjustment to the pose of the display device.

In a second aspect, a navigation system of the system of the firstaspect is provided.

In a third aspect, a non-transitory computer readable medium or computerprogram product are provided comprising instructions, which whenexecuted by one or more processors, are configured to implement thecontrol system of the first aspect.

In a fourth aspect, a method of operating a system for aiding ininteraction with a physical object is provided, the system comprising: adisplay device defining a plane and being located on a first side of thephysical object; and a navigation system coupled to a control system,the method comprising: registering computer images to the physicalobject; tracking a pose of the physical object and the display device ina common coordinate system; and controlling the display device to renderthe registered computer images according to the tracked pose of thephysical object and the display device; and wherein a perspective of therendering is based on a virtual camera having a virtual position locatedon a second side of the physical object opposite to the first side, andwherein the virtual camera has a field-of-view facing the plane ofdisplay device, and automatically updating the virtual position of thevirtual camera in response to adjusting the pose of the display device.

In a fifth aspect, a non-transitory computer readable medium or computerprogram product are provided comprising instructions, which whenexecuted by one or more processors, are configured to implement themethod of the fourth aspect.

In a sixth aspect, a system for aiding in interaction with a physicalobject is provided, the system comprising: a display device defining aplane, wherein the physical object is located in front of the plane; anavigation system coupled to a control system and being configured to:register computer images to the physical object; track a pose of thephysical object, the display device, and a user's viewpoint in a commoncoordinate system; and control the display device for rendering theregistered computer images according to the tracked pose of the physicalobject, the display device, and the user's viewpoint; and wherein aperspective of the rendering is based on a field-of-view of a virtualcamera that has a virtual position behind the plane, and wherein thevirtual position of the virtual camera is automatically updated inresponse to movement of the tracked pose of the user's viewpoint.

In a seventh aspect, a navigation system of the system of the sixthaspect is provided.

In an eighth aspect, a non-transitory computer readable medium orcomputer program product are provided comprising instructions, whichwhen executed by one or more processors, are configured to implement thecontrol system of the sixth aspect.

In a ninth aspect, a method of operating a system for aiding ininteraction with a physical object is provided, the system comprising adisplay device defining a plane, wherein the physical object is locatedin front of the plane, and a navigation system coupled to a controlsystem, the method comprising: registering computer images to thephysical object; tracking a pose of the physical object, the displaydevice, and a user's viewpoint in a common coordinate system; andcontrolling the display device for rendering the registered computerimages according to the tracked pose of the physical object, the displaydevice, and the user's viewpoint; and wherein a perspective of therendering is based on a field-of-view of a virtual camera having avirtual position behind the plane, and automatically updating thevirtual position of the virtual camera in response to movement of thetracked pose of the user's viewpoint.

In a tenth aspect, a non-transitory computer readable medium or computerprogram product are provided comprising instructions, which whenexecuted by one or more processors, are configured to implement themethod of the ninth aspect.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example of a surgical system that can be utilizedwith the spatially-aware display techniques described herein.

FIG. 2 is a simulated illustration of one implementation and setup of aFixed View Frustum visualization technique wherein the perspective ofthe images displayed depend on the pose of the display device and poseof the physical object (e.g., patient) relative to the display device.

FIG. 3A is an illustrative real-world example taken from a first-personperspective of one implementation of the Fixed View Frustumvisualization technique illustrating a user manually adjusting thedisplay device to adjust the visualization of a 3D model of the physicalobject.

FIGS. 3B and 3C provide simulated illustrations of one implementation ofthe Fixed View Frustum visualization technique comparatively showingadjustment to the pose of the display device and virtual camerapositions.

FIGS. 4A and 4B are diagrams illustrating implementation of the ScreenParallel Slice Visualization technique wherein slices of the 3D model ofthe object are obtained and displayed on the display device in parallelwith the plane of the display device.

FIGS. 5A and 5B are illustrative real-world examples, taken from a firstperson perspective, of one implementation of the Screen Parallel SliceVisualization technique utilized in conjunction with the Fixed ViewFrustum visualization technique comparatively showing the user changingthe pose of the display device which causes the slice to changeperspective correspondingly.

FIG. 6 is an illustrative real-world example of one implementation ofthe Screen Parallel Slice Visualization technique wherein the slice isdisplayed relative to the 3D model of the physical object (e.g.,patient) depending on the pose of a tracked tool.

FIG. 7 is an illustrative real-world example of one implementation ofthe Screen Parallel Slice Visualization technique wherein the displayedimages are modified to account for the spatial layering of the object,tool, and slice relative to one other so as to mimic the actual spatiallocation of the same in the real coordinate system in which these itemsexist.

FIG. 8 is one implementation of an adjustment system that is configuredto adjust the pose of the display device, for example, for use with theFixed View Frustum visualization technique.

FIG. 9 is a simulated illustration of one implementation and setup of aDynamic Mirror View Frustum visualization technique wherein theperspective of the images displayed depend on the pose of the displaydevice, the pose of the object and the pose of a user's viewpoint toenable the displayed images to be rotated and sized to match the user'spoint of view.

FIGS. 10A and 10C provide simulated illustrations of one implementationof the Dynamic Mirror View Frustum visualization technique comparativelyshowing from a 3^(rd) person view changes to the pose of the respectiveuser viewpoint and virtual camera positions.

FIGS. 10B and 10D illustrate the environment of FIGS. 10A and 10C,respectively, as taken from a 1^(st) person perspective.

FIGS. 10E and 10F are illustrative real-world examples of oneimplementation of the Dynamic Mirror View Frustum visualizationtechnique, taken from 1^(st) person perspective, and comparativelyshowing changes to the user's perspective relative to the display devicecausing the displayed images to be rotated and sized to match the user'spoint of view.

FIGS. 11A-11C are diagrams illustrating, in various views of the X,Y,Zcoordinate system, one implementation Dynamic Mirror View Frustumvisualization technique comparatively showing, for three scenarios,changes to the pose of the respective user viewpoint and virtual camerapositions and rotation of the image rendering to align to the user'sperspective.

FIGS. 12A and 12B are illustrative real-world examples, taken from a1^(st) person perspective, of one implementation of a Viewpoint FacingSlice Visualization technique utilized in conjunction with the DynamicMirror View Frustum visualization technique comparatively showing thehow changing the user's viewpoint relative to the display device causesthe displayed slice to change perspective to align to the user'sviewpoint.

DETAILED DESCRIPTION

I. Surgical System Overview

Referring to FIG. 1, a surgical system 10 is illustrated which can beutilized with the spatially-aware displays described in Section IIbelow. The system 10 is useful for treating a target site or anatomicalvolume A of a patient 12, such as treating bone or soft tissue. In FIG.1, the patient 12 is undergoing a surgical procedure. The anatomy inFIG. 1 includes a femur F, pelvis PEL, and a tibia T of the patient 12.The surgical procedure may involve tissue removal or other forms oftreatment. Treatment may include cutting, coagulating, lesioning thetissue, other in-situ tissue treatments, or the like. In some examples,the surgical procedure involves partial or total knee or hip replacementsurgery, shoulder replacement surgery, spine surgery, or ankle surgery.In some examples, the system 10 is designed to cut away material to bereplaced by surgical implants, such as hip and knee implants, includingunicompartmental, bicompartmental, multicompartmental, or total kneeimplants, acetabular cup implants, femur stem implants, screws, anchors,other fasteners, and the like. Some of these types of implants are shownin U.S. Patent Application Publication No. 2012/0330429, entitled,“Prosthetic Implant and Method of Implantation,” the disclosure of whichis hereby incorporated by reference. The system 10 and techniquesdisclosed herein may be used to perform other procedures, surgical ornon-surgical, or may be used in industrial applications or otherapplications.

The system 10 may include a robotic manipulator 14, also referred to asa surgical robot. The manipulator 14 has a base 16 and plurality oflinks 18. A manipulator cart 17 supports the manipulator 14 such thatthe manipulator 14 is fixed to the manipulator cart 17. The links 18collectively form one or more arms of the manipulator 14 (e.g., roboticarms). The manipulator 14 may have a serial arm configuration (as shownin FIG. 1), a parallel arm configuration, or any other suitablemanipulator configuration. In other examples, more than one manipulator14 may be utilized in a multiple arm configuration.

In the example shown in FIG. 1, the manipulator 14 comprises a pluralityof joints J and a plurality of joint encoders 19 located at the joints Jfor determining position data of the joints J. For simplicity, only onejoint encoder 19 is illustrated in FIG. 1, although other joint encoders19 may be similarly illustrated. The manipulator 14 according to oneexample has six joints J1-J6 implementing at least six-degrees offreedom (DOF) for the manipulator 14. However, the manipulator 14 mayhave any number of degrees of freedom and may have any suitable numberof joints J and may have redundant joints.

The manipulator 14 need not require joint encoders 19 but mayalternatively, or additionally, utilize motor encoders present on motorsat more than one or each joint J. Also, the manipulator 14 need notrequire rotary joints, but may alternatively, or additionally, utilizeone or more prismatic joints. Any suitable combination of joint types iscontemplated.

The base 16 of the manipulator 14 is generally a portion of themanipulator 14 that provides a fixed reference coordinate system forother components of the manipulator 14 or the system 10 in general.Generally, the origin of a manipulator coordinate system MNPL is definedat the fixed reference of the base 16. The base 16 may be defined withrespect to any suitable portion of the manipulator 14, such as one ormore of the links 18. Alternatively, or additionally, the base 16 may bedefined with respect to the manipulator cart 17, such as where themanipulator 14 is physically attached to the cart 17. In one example,the base 16 is defined at an intersection of the axes of joints J1 andJ2. Thus, although joints J1 and J2 are moving components in reality,the intersection of the axes of joints J1 and J2 is nevertheless avirtual fixed reference pose, which provides both a fixed position andorientation reference and which does not move relative to themanipulator 14 and/or manipulator cart 17.

In some examples, the manipulator 14 can be a hand-held manipulatorwhere the base 16 is a base portion of a tool (e.g., a portion heldfree-hand by the user) and the tool tip is movable relative to the baseportion. The base portion has a reference coordinate system that istracked and the tool tip has a tool tip coordinate system that iscomputed relative to the reference coordinate system (e.g., via motorand/or joint encoders and forward kinematic calculations). Movement ofthe tool tip can be controlled to follow the path since its poserelative to the path can be determined. Such a manipulator 14 is shownin U.S. Pat. No. 9,707,043, filed on Aug. 31, 2012, entitled, “SurgicalInstrument Including Housing, A Cutting Accessory that Extends from theHousing and Actuators that Establish the Position of the CuttingAccessory Relative to the Housing,” which is hereby incorporated hereinby reference.

The manipulator 14 and/or manipulator cart 17 house a manipulatorcontroller 26, or other type of control unit. The manipulator controller26 may comprise one or more computers, or any other suitable form ofcontroller that directs the motion of the manipulator 14. Themanipulator controller 26 may have a central processing unit (CPU)and/or other processors, memory (not shown), and storage (not shown).The manipulator controller 26 is loaded with software as describedbelow. The processors could include one or more processors to controloperation of the manipulator 14. The processors can be any type ofmicroprocessor, multi-processor, and/or multi-core processing system.The manipulator controller 26 may additionally, or alternatively,comprise one or more microcontrollers, field programmable gate arrays,systems on a chip, discrete circuitry, and/or other suitable hardware,software, or firmware that is capable of carrying out the functionsdescribed herein. The term processor is not intended to limit anyimplementation to a single processor. The manipulator 14 may alsocomprise a user interface UI with one or more displays and/or inputdevices (e.g., push buttons, sensors, switches, keyboard, mouse,microphone (voice-activation), gesture control devices, touchscreens,joysticks, foot pedals, etc.).

A surgical tool 20 couples to the manipulator 14 and is movable relativeto the base 16 to interact with the anatomy in certain modes. The tool20 is or forms part of an end effector 22 supported by the manipulator14 in certain embodiments. The tool 20 may be grasped by the user. Onepossible arrangement of the manipulator 14 and the tool 20 is describedin U.S. Pat. No. 9,119,655, filed on Aug. 2, 2013, entitled, “SurgicalManipulator Capable of Controlling a Surgical Instrument in MultipleModes,” the disclosure of which is hereby incorporated by reference. Themanipulator 14 and the tool 20 may be arranged in alternativeconfigurations. The tool 20 can be like that shown in U.S. PatentApplication Publication No. 2014/0276949, filed on Mar. 15, 2014,entitled, “End Effector of a Surgical Robotic Manipulator,” herebyincorporated by reference. Separate hand-held surgical tools can beutilized in addition to or alternative to the manipulator 14 and tool20.

The tool 20 includes an energy applicator 24 designed to contact thetissue of the patient 12 at the target site. In one example, the energyapplicator 24 is a bur 25. The bur 25 may be spherical and comprise aspherical center, radius (r) and diameter. Alternatively, the energyapplicator 24 may be a drill bit, a saw blade 27 (see alternative toolin FIG. 1), an ultrasonic vibrating tip, or the like. The tool 20 and/orenergy applicator 24 may comprise any geometric feature, e.g.,perimeter, circumference, radius, diameter, width, length, volume, area,surface/plane, range of motion envelope (along any one or more axes),etc. The geometric feature may be considered to determine how to locatethe tool 20 relative to the tissue at the target site to perform thedesired treatment. In some of the embodiments described herein, aspherical bur having a tool center point (TCP) and a sagittal saw bladehaving a TCP will be described for convenience and ease of illustration,but is not intended to limit the tool 20 to any particular form.

The tool 20 may comprise a tool controller 21 to control operation ofthe tool 20, such as to control power to the tool 20 (e.g., to a tooldrive such as a rotary motor of the tool 20), control movement of thetool 20, control irrigation/aspiration of the tool 20, and/or the like.The tool controller 21 may be in communication with the manipulatorcontroller 26 or other components. The tool 20 may also comprise a userinterface UI with one or more displays and/or input devices (e.g., pushbuttons, triggers, sensors, switches, keyboard, mouse, microphone(voice-activation), gesture control devices, touchscreens, joysticks,foot pedals, etc.) that are coupled to the tool controller 21,manipulator controller 26, and/or other controllers described herein.The manipulator controller 26 controls a state (e.g., position and/ororientation) of the tool 20 (e.g., of the TCP) with respect to acoordinate system, such as the manipulator coordinate system MNPL. Themanipulator controller 26 can control velocity (linear or angular),acceleration, or other derivatives of motion of the tool 20.

The tool center point (TCP), in one example, is a predeterminedreference point defined at the energy applicator 24. The TCP has aknown, or able to be calculated (i.e., not necessarily static), poserelative to other coordinate systems. The geometry of the energyapplicator 24 is known in or defined relative to a TCP coordinatesystem. The TCP may be located at the spherical center of the bur 25 ofthe tool 20 or at the distal end of the saw blade 27 such that only onepoint is tracked. The TCP may be defined in various ways depending onthe configuration of the energy applicator 24. The manipulator 14 couldemploy the joint/motor encoders, or any other non-encoder positionsensing method, to enable a pose of the TCP to be determined. Themanipulator 14 may use joint measurements to determine TCP pose and/orcould employ techniques to measure TCP pose directly. The control of thetool 20 is not limited to a center point. For example, any suitableprimitives, meshes, etc., can be used to represent the tool 20.

The system 10 further includes a navigation system 32. One example ofthe navigation system 32 is described in U.S. Pat. No. 9,008,757, filedon Sep. 24, 2013, entitled, “Navigation System Including Optical andNon-Optical Sensors,” hereby incorporated by reference. The navigationsystem 32 tracks movement of various objects. Such objects include, forexample, the manipulator 14, the tool 20 and the anatomy, e.g., femur F,pelvis PEL, and tibia T. The navigation system 32 tracks these objectsto gather state information of the objects with respect to a(navigation) localizer coordinate system LCLZ. Coordinates in thelocalizer coordinate system LCLZ may be transformed to the manipulatorcoordinate system MNPL, to other coordinate systems, and/or vice-versa,using transformations.

The navigation system 32 may include a cart assembly 34 that houses anavigation controller 36, and/or other types of control units. Anavigation user interface UI is in operative communication with thenavigation controller 36. The navigation user interface includes one ormore displays 38. The navigation system 32 is capable of displaying agraphical representation of the relative states of the tracked objectsto the user using the one or more displays 38. The navigation userinterface UI further comprises one or more input devices to inputinformation into the navigation controller 36 or otherwise toselect/control certain aspects of the navigation controller 36. Suchinput devices include interactive touchscreen displays. However, theinput devices may include any one or more of push buttons, a keyboard, amouse, a microphone (voice-activation), gesture control devices, footpedals, and the like.

The navigation system 32 also includes a navigation localizer 44 coupledto the navigation controller 36. In one example, the localizer 44 is anoptical localizer and includes a camera unit 46. The camera unit 46 hasan outer casing 48 that houses one or more optical sensors 50. Thelocalizer 44 may comprise its own localizer controller 49 and mayfurther comprise a video camera VC.

The navigation system 32 includes one or more trackers. In one example,the trackers include a pointer tracker PT, one or more manipulatortrackers 52A, 52B, a first patient tracker 54, a second patient tracker55, and a third patient tracker 56. In the illustrated example of FIG.1, the manipulator tracker is firmly attached to the tool 20 (i.e.,tracker 52A), the first patient tracker 54 is firmly affixed to thefemur F of the patient 12, the second patient tracker 55 is firmlyaffixed to the pelvis PEL of the patient 12, and the third patienttracker 56 is firmly affixed to the tibia T of the patient 12. In thisexample, the patient trackers 54, 55, 56 are firmly affixed to sectionsof bone. The pointer tracker PT is firmly affixed to a pointer P usedfor registering the anatomy to the localizer coordinate system LCLZ. Themanipulator tracker 52A, 52B may be affixed to any suitable component ofthe manipulator 14, in addition to, or other than the tool 20, such asthe base 16 (i.e., tracker 52B), or any one or more links 18 of themanipulator 14. The trackers 52A, 52B, 54, 55, 56, PT may be fixed totheir respective components in any suitable manner. For example, thetrackers may be rigidly fixed, flexibly connected (optical fiber), orphysically spaced (e.g., ultrasound), as long as there is a suitable(supplemental) way to determine the relationship (measurement) of thatrespective tracker to the object with which it is associated.

Any one or more of the trackers may include active markers 58. Theactive markers 58 may include light emitting diodes (LEDs).Alternatively, the trackers 52A, 52B, 54, 55, 56, PT may have passivemarkers, such as reflectors, which reflect light emitted from the cameraunit 46. Other suitable markers not specifically described herein may beutilized.

The localizer 44 tracks the trackers 52A, 52B, 54, 55, 56, PT todetermine a state of the trackers 52A, 52B, 54, 55, 56, PT, whichcorrespond respectively to the state of the object respectively attachedthereto. The localizer 44 may perform known triangulation techniques todetermine the states of the trackers 52, 54, 55, 56, PT, and associatedobjects. The localizer 44 provides the state of the trackers 52A, 52B,54, 55, 56, PT to the navigation controller 36. In one example, thenavigation controller 36 determines and communicates the state thetrackers 52A, 52B, 54, 55, 56, PT to the manipulator controller 26. Asused herein, the state of an object includes, but is not limited to,data that defines the position and/or orientation of the tracked objector equivalents/derivatives of the position and/or orientation. Forexample, the state may be a pose of the object, and may include linearvelocity data, and/or angular velocity data, and the like.

The navigation controller 36 may comprise one or more computers, or anyother suitable form of controller. Navigation controller 36 has acentral processing unit (CPU) and/or other processors, memory (notshown), and storage (not shown). The processors can be any type ofprocessor, microprocessor or multi-processor system. The navigationcontroller 36 is loaded with software. The software, for example,converts the signals received from the localizer 44 into datarepresentative of the position and orientation of the objects beingtracked. The navigation controller 36 may additionally, oralternatively, comprise one or more microcontrollers, field programmablegate arrays, systems on a chip, discrete circuitry, and/or othersuitable hardware, software, or firmware that is capable of carrying outthe functions described herein. The term processor is not intended tolimit any implementation to a single processor.

Although one example of the navigation system 32 is shown that employstriangulation techniques to determine object states, the navigationsystem 32 may have any other suitable configuration for tracking themanipulator 14, tool 20, and/or the patient 12. In another example, thenavigation system 32 and/or localizer 44 are ultrasound-based. Forexample, the navigation system 32 may comprise an ultrasound imagingdevice coupled to the navigation controller 36. The ultrasound imagingdevice images any of the aforementioned objects, e.g., the manipulator14, the tool 20, and/or the patient 12, and generates state signals tothe navigation controller 36 based on the ultrasound images. Theultrasound images may be 2-D, 3-D, or a combination of both. Thenavigation controller 36 may process the images in near real-time todetermine states of the objects. The ultrasound imaging device may haveany suitable configuration and may be different than the camera unit 46as shown in FIG. 1.

In another example, the navigation system 32 and/or localizer 44 areradio frequency (RF)-based. For example, the navigation system 32 maycomprise an RF transceiver coupled to the navigation controller 36. Themanipulator 14, the tool 20, and/or the patient 12 may comprise RFemitters or transponders attached thereto. The RF emitters ortransponders may be passive or actively energized. The RF transceivertransmits an RF tracking signal and generates state signals to thenavigation controller 36 based on RF signals received from the RFemitters. The navigation controller 36 may analyze the received RFsignals to associate relative states thereto. The RF signals may be ofany suitable frequency. The RF transceiver may be positioned at anysuitable location to track the objects using RF signals effectively.Furthermore, the RF emitters or transponders may have any suitablestructural configuration that may be much different than the trackers52A, 52B, 54, 55, 56, PT shown in FIG. 1.

In yet another example, the navigation system 32 and/or localizer 44 areelectromagnetically based. For example, the navigation system 32 maycomprise an EM transceiver coupled to the navigation controller 36. Themanipulator 14, the tool 20, and/or the patient 12 may comprise EMcomponents attached thereto, such as any suitable magnetic tracker,electro-magnetic tracker, inductive tracker, or the like. The trackersmay be passive or actively energized. The EM transceiver generates an EMfield and generates state signals to the navigation controller 36 basedupon EM signals received from the trackers. The navigation controller 36may analyze the received EM signals to associate relative statesthereto. Again, such navigation system 32 examples may have structuralconfigurations that are different than the navigation system 32configuration shown in FIG. 1.

The navigation system 32 may have any other suitable components orstructure not specifically recited herein. Furthermore, any of thetechniques, methods, and/or components described above with respect tothe navigation system 32 shown may be implemented or provided for any ofthe other examples of the navigation system 32 described herein. Forexample, the navigation system 32 may utilize solely inertial trackingor any combination of tracking techniques, and may additionally oralternatively comprise, fiber optic-based tracking, machine-visiontracking, and the like. While in our certain implementation, optical IRtracking is utilized, the concepts and techniques described herein maybe utilized to work with any sufficiently accurate 6D trackingtechnology. Furthermore, we assume that existing surgical navigationsystems already contain appropriate methods for registeringpre-operative image data and surgical plans to the patient beforesurgery.

The system 10 includes a control system 60 that comprises, among othercomponents, any one or more of the manipulator controller 26, thenavigation controller 36, and the tool controller 21. The control system60 includes one or more software programs and software modules. Thesoftware modules may be part of the program or programs that operate onthe manipulator controller 26, navigation controller 36, tool controller21, or any combination thereof, to process data to assist with controlof the system 10. The software programs and/or modules include computerreadable instructions stored in non-transitory memory 64 on themanipulator controller 26, navigation controller 36, tool controller 21,or a combination thereof, to be executed by one or more processors 70 ofthe controllers 21, 26, 36. The memory 64 may be any suitableconfiguration of memory, such as RAM, non-volatile memory, etc., and maybe implemented locally or from a remote database. Additionally, softwaremodules for prompting and/or communicating with the user may form partof the program or programs and may include instructions stored in memory64 on the manipulator controller 26, navigation controller 36, toolcontroller 21, or any combination thereof. The user may interact withany of the input devices of the navigation user interface UI or otheruser interface UI to communicate with the software modules. The userinterface software may run on a separate device from the manipulatorcontroller 26, navigation controller 36, and/or tool controller 21.

The control system 60 may comprise any suitable configuration of input,output, and processing devices suitable for carrying out the functionsand methods described herein. The control system 60 may comprise themanipulator controller 26, the navigation controller 36, or the toolcontroller 21, or any combination thereof, or may comprise only one ofthese controllers. These controllers may communicate via a wired bus orcommunication network, via wireless communication, or otherwise. Thecontrol system 60 may also be referred to as a controller. The controlsystem 60 may comprise one or more microcontrollers, field programmablegate arrays, systems on a chip, discrete circuitry, sensors, displays,user interfaces, indicators, and/or other suitable hardware, software,or firmware that is capable of carrying out the functions describedherein.

II. Spatially-Aware Display Techniques

Described herein are systems, methods, and techniques related tospatially-aware displays for computer assisted interventions. A noveldisplay and visual interaction paradigm is presented, which aims atreducing the complexity of understanding the spatial transformationsbetween the user's (e.g., surgeon) viewpoint, a physical object (e.g., apatient), 2D and 3D data (e.g., the pre/intra-operative patient data),and tools during computer-assisted interventions. The interventionaldisplay, for example in surgical navigation systems, can be registeredboth to the patient and to the surgeon's view. With this technique, thesurgeon can keep his/her own direct view to the patient independent ofany need for additional display or direct view augmentation. In someimplementations, the monitor used in the operating room is registered tothe patient and surgeon's viewpoint. This enables the physicians toeffortlessly relate their view of tools and patient to the virtualrepresentation of the patient data. The direct view of the surgeon ontothe patient and his/her working space can remain unchanged. The positionand orientation of the display plays an integral part of thevisualization pipeline. Therefore, the pose of the display isdynamically tracked relative to other objects of interest, such as thepatient, instruments, and in some implementations, the surgeon's head.This information is then used as an input to the image-guided surgeryvisualization user interface.

At least two implementations are presented. The first one uses a “FixedView Frustum” relating the pose of the display to the patient and tools.For the Fixed View Frustum technique, the display is tracked for exampleby attaching a tracking marker and calibrating the spatial relationbetween the physical display and the tracking marker.

The second one is built upon the mirror metaphor and extends the firsttechnique to also integrate the pose of the surgeon's head as anadditional parameter within the visualization pipeline, in which casethe display will be associated to a “Dynamic Mirror View Frustum”. Thesurgeon's viewpoint is tracked for visualization techniques for theDynamic Mirror View Frustum. Estimation of the surgeon's viewpoint canbe achieved either by using a head tracking target or by mounting acamera to the surgical display in combination with existing video-basedhead-pose estimation algorithms. Once the tracking information isavailable in a common global co-ordinate system, we can compute thespatial relationship between the patient, the surgical display, thetools and the surgeon's viewpoint by deriving the relevanttransformations from the spatial relations of the tracked entities.

These novel display and visual exploration paradigms aim at reducing thecomplexity of understanding spatial transformations between a user'sviewpoint, the physical object (O), the pre/intra-operative 2D and 3Ddata, and surgical tools 110, 20 during computer assisted interventionswith minimal change in the current setups. Any surgical tracking systemcan be used to track the display, tool and user's head supporting theintegration into computer assisted intervention systems. The solutionspresented allow physicians to effortlessly relate their view of toolsand the patient to the virtual data on surgical monitors. The users gainthe possibility of interacting with the patient data just by intuitivelymoving their viewing position and observing it from a differentperspective in relation to the patient position independent from theneed for an interaction device like a mouse or joystick.

A. Fixed View Frustum Visualization Technique

With reference to FIGS. 2-8, one example visualization method that canbe utilized with the surgical system 10 comprises a Fixed View Frustum(FVF) for aiding in user interaction with a physical object (O). Thephysical object (O) can be a physical anatomy, as shown in the Figures,or any other object requiring interaction, setup, or intervention, suchas a surgical device, robotic device, surgical training model of ananatomy, and the like. For simplicity, the physical anatomy is shown asthe object, however, the concept is not limited to such.

One or more external screen(s) or display device(s) 100 are provided.These display devices 100 can be those displays 38 on the navigationcart 34 assembly or any other external display that is spaced apart fromand not worn by the user. The display device 100 can be any suitabletype of display, including, but not limited to: LED, LCD, OLED,touchscreen, holographic; and can display any type of imagery. Thedisplay device 100 can also take any suitable geometric shape, includingrectangular, square, circular, or the like.

During use, the display device (s) 100 is located on a side (S1) of thephysical object (O) opposite to a side (S2) where the user is located.In other words, the physical object (O) is between a user's viewpoint(V) and the display device 100. Hence, the display device 100 utilizedherein is distinguished from head-mounted displays or tablet screensthat are between the user's viewpoint (V) and the object (O). Here, thephysical object (O) is shown as a virtual representation merely forillustrative purposes.

Of course, the user is not prohibited from moving between the physicalobject (O) and the display device 100. However, implementation of thespatially-aware display takes into account the practical reality that asurgeon desires to visualize the physical object (O) on the side whichhe/she is presently located and that typically the side of the physicalobject (O) that is opposite to the surgeon would provide an invertedperspective. This will be understood further below in context of thefield of view of the virtual camera which forward-faces the displaydevice 100.

The display device 100 defines a plane (P). The plane (P) is definedparallel to or coincident with the actual front face of the displaydevice 100. Here, the plane (P) is a virtual object utilized forcomputational purposes, as will be described below. The control system60, which includes the navigation system 32 comprising any one or morecontrollers described herein, is configured to register computer images(R) to the physical object (O). Registration of the physical object (O)is not limited to any technique and can occur according to any suitablemethod including digitization of the physical object (O), touchlessregistration (e.g., ultrasonic), 2D/3D image registration utilizing animaging device (e.g., CT, X-Ray), or the like. The computer images (R)can be represented as a virtual model (VM) of any portions of thephysical object (O). These computer images (R) can be rendered on thedisplay device 100. These computer images (R) can be static or dynamicand can include actual/real video graphic images, mixed reality,augmented reality, virtual reality images or models, or any combinationthereof.

The control system 60, by way of the localizer 44 tracks a pose of thephysical object (O) utilizing any of the patient tracking techniquesdescribed above, including but not limited to the patient trackers 54,55, 56. In some implementations, the localizer 44 can also track a poseof the display device 100 in a common coordinate system with thephysical object (O). The display device 100 can be tracked using adisplay tracker 102, or by any other suitable technique such as thosedescribed above. For instance, the display device 100 can be trackedusing machine vision (e.g., with or without trackers), tracked usingoptical or non-optical techniques. Alternatively, or additionally, whenequipped with a motorized adjustment system, the pose of the displaydevice 100 can be determined based on kinematic data, and independent ofthe localizer 44, as will be described below.

The control system 60 controls the display device 100 to render theregistered computer images (R) according to at least the tracked pose ofthe physical object (O) and the display device 100. Hence, the renderingof the computer images (R) is dependent upon at least the pose thephysical object (O) and the pose of the display device 100.Specifically, referring to FIG. 2, a perspective of the rendering isbased on a point or field of view (FOV) of a virtual camera (VC) thatfaces the plane (P) of display device 100. The field of view (FOV) canbe based on a pinhole camera model projected towards the display device100. The field of view (FOV) is also known as the viewing frustum (F),which will be described below.

A virtual position (VP) of the virtual camera (VC) is located on a side(S1) of the physical object (0) that is opposite from the side (S2) ofthe display device 100. Since the virtual camera (VC) is located infront of the plane (P) of the display device 100 in FIG. 2, the computerimage (R) is showing the side of the physical object (O) at side (S1),rather than a mirrored view at side (S2).

The virtual position (VP) may be located at a predetermined distance (d)from the plane (P) of display device 100. The distance (d) is based on aZ-axis line drawn between the virtual position (VP) of the virtualcamera (VC) and the plane (P). The Z-axis may be defined with referenceto the virtual position (VP) or the display device 100. The virtualposition (VP) can be anywhere on the virtual camera (VC) and can bedefined by a single point or points, or any type of surface orvolumetric geometry. In some examples, the virtual position (VP) remainsfixed at a predetermined distance (d). Alternatively, the predetermineddistance (d) can change in response to certain conditions. For example,the user can specify to the control system 60 a preference for thedistance (d). The distance (d) can change automatically depending uponconditions such as, but not limited to: a type of surgical procedure, acertain step of the procedure, a tracked pose of the user's viewpoint(V), a tracked pose of the object, a tracked pose of the display device100, a tracked pose of a tool 110, 20, or the like. In one example, thedistance (d) can be defined at any suitable distance and within anysuitable range, including, but not limited to between 1 meter and 5meters, or any distance inclusively therebetween.

In one example, the virtual position (VP) of the virtual camera (VC) istransverse to the plane (P) of the display device 100. In other words, aZ-axis line drawn between the virtual position (VP) and the plane (P) ofthe display device 100 defines an angle relative to the plane (P)wherein the angle is in a range between 0 and 180 degrees. In a morespecific implementation, the virtual position (VP) of the virtual camera(VC) is orthogonal to the plane (P) of the display device 100. In otherwords, a line drawn between the virtual position (VP) and the plane (P)of the display device 100 is 90 degrees normal to the plane (P). Inother words, in this example, computer images (R) are rendered with anon-axis perspective projection orthogonal to the display device 100 fromthe given distance (d).

The virtual position (VP) of the virtual camera (VC) also has X and Ycoordinate locations relative to the plane (P) of the display device100. The X and Y coordinates may be defined with reference to thevirtual position (VP) or the display device 100. In one example, the Xand Y coordinates are defined in a plane that is parallel to the plane(P) of the display device 100. However, the X and Y coordinate planeneed not be parallel to the plane (P), for example, if the virtualcamera (VC) is projecting towards the display device 100 at anon-orthogonal angle.

Furthermore, whether projecting transverse or orthogonal to the displaydevice 100, the line drawn between the virtual position (VP) and theplane (P) may be directed towards the geometrical center of the displaydevice 100. Alternatively, the line drawn between the virtual position(VP) and the plane (P) may be offset from geometrical center of thedisplay device 100. For example, the offset location can be towards acorner or edge of the display device 100. The X-Y placement of virtualposition (VP) relative to the display device 100 can change in responseto certain conditions. For example, the user can specify to the controlsystem 60 a preference for X-Y placement of virtual position (VP). TheX-Y placement of virtual position (VP) can change automaticallydepending upon conditions such as, but not limited to: a type ofsurgical procedure, a certain step of the procedure, a tracked pose ofthe user's viewpoint (V), a tracked pose of the object, a tracked poseof the display device 100, a tracked pose of a surgical tool 20, or thelike. In one example, the distance (d) can be defined at any suitabledistance and within any suitable range, including, but not limited tobetween 1 meter and 5 meters, or any distance inclusively therebetween.

Rendering of the computer images (R) based on the virtual camera (VC) isimplemented using a viewing frustum (F) originating from the virtualcamera (VC) and passing through an object, which in this case is theplane (P) of the display device 100. The viewing frustum (F) may be aregion of space in a 3D modeled world that may appear on the displaydevice 100 and may considered as the field of view of the virtual camera(VC). An apex of the frustum (F) is the zero-point originating from thevirtual camera (VC) and may also be regarded as the virtual position(VP) of the virtual camera (VC). In one example, the plane (P) of thedisplay device 100 may be located at a base (B) or far-clipping plane(FCP) of the viewing frustum (F). The viewing frustum (F) may comprise anear clipping plane (NCP) proximate to the virtual camera (VC). Thefar-clipping plane (FCP) and near-clipping planes (NCP) virtually cutthe frustum (F) perpendicular to the viewing direction such that objectscloser to the camera than the near-clipping planes (NCP) or beyond thefar-clipping plane (FCP) are not rendered on the computer image (R).Objects that lie partially or completely outside of the viewing frustum(F) can be removed from the rendering process for processing efficiency.The computer images (R) on the display device 100 are projected based onthe viewing frustum (F). The viewing frustum (F) can be based off anyplane truncation or any suitable 3-shape, including a pyramid, cone, orthe like. The viewing frustum (F) can take any configuration other thanthat shown in the Figures or described herein.

In the FVF technique, the virtual position (VP) of the virtual camera(VC) is automatically updated by the control system 60 in response to(manual or motorized) adjustment to the pose of the display device 100.If the user moves or rotates the display device 100, the position of thevirtual camera (VC) is automatically updated. Therefore, in contrast tothe standard visualization displays within surgical navigation systems,the mental mapping of the real object to its image on the screen issimplified.

Examples of such Fixed View Frustum display visualizations are shown inFIGS. 3A-3C. An illustrative real-world example is shown in FIG. 3A andtwo simulations are shown in FIGS. 3B and 3C, respectively. FIG. 3Ashows an operator moving the display device 100 by hand to visualize theinternal virtual model (VM) of the physical object (O). As the useradjusts the display device 100, the computer rendering (R) on thedisplay device 100 updates accordingly. FIGS. 3B and 3C providesimulation illustrations of the FVF technique from two differentviewpoints of the same configuration of the virtual camera (VC). Ascomparatively shown, the display device 100 changes pose between FIGS.3B and 3C and the virtual position (VP) of the virtual camera (VC)updates accordingly. The computer image (R) also changes perspective inaccordance with the relative positioning between the viewing frustum (F)and the physical object (O).

i. Screen Parallel Slice Visualization (SPSV)

Referring now to FIGS. 4-6, a Screen Parallel Slice Visualization (SPSV)sub-technique of the FVF method is described. While the FVFvisualization is intuitive to use and can display 3D data, the techniquecan be further implemented using slice visualization. Accordingly, onecan incorporate the slice view into the spatially-aware visualizationconcept described.

With reference to FIG. 4A, this can be achieved in one implementation byslicing a 3D virtual model (VM) of the physical object (O) into aplurality of slices (SL1, SL2, SL3 . . . SLN). In one implementation,the slices (SL) are made at planes parallel to the plane (P) of thedisplay device 100. The virtual model (VM) can be a CT model, X-raymodel, MRI model or a model created using any other type of imagingtechnique. Alternatively, instead of slicing a 3D virtual model (VM),the imaging data of the physical object (O) may comprise a plurality ofslices that can be obtained from computer-memory with or without havingcombined the same into a 3D model.

In one implementation, and with reference to FIG. 4B, the displayedslice (SL) can be based on a designated plane (p) sliced through theviewing frustum (F). The designated plane (p) can be fixed relative tothe virtual camera (VC) position, or the designated plane (p) candynamically change based on any of the conditions described herein. Whenthe field of view of the virtual camera (VC) passes through the physicalobject (O), the control system 60 can immediately obtain the slice (SL)that corresponds to the intersection of the designated plane (p) withthe physical object (O).

The orientation of the computer image (R) of the slice (SL) can beadjusted based on the pose of the display device 100. As showncomparatively between FIGS. 5A and 5B, the user is given the interactionmethod to rotate the display device 100, which in turn also rotates theslice (SL) and a virtual model (T) of the tool 110, 20. The physicalobject (O) and tool 110, 20 are in the same relative position in FIGS.5A and 5B.

Alternatively, or additionally, as shown in FIG. 6, the orientation ofthe slice (SL) can be set based on a tool 110, 20 brought within theviewing frustum (F) of the virtual camera (VC). A pose of the tool 110,20 can be tracked by the localizer 44 using any suitable means, such asthose described herein. The computer images (R) that are rendered caninclude images of the tool 110, 20 and slice (SL) using the Fixed ViewFrustum method as discussed above. Alternatively, or additionally, theorientation of the slice (SL) can be set based on a user's perspectiveas defined by a tracked pose of a viewpoint tracking system (VTS), suchas an external camera (C) facing the user or a head-mounted device 120(described in detail below).

The displayed slice (SL) can be dynamically changed to different slices(SL) during use of these techniques. The slice (SL) can change dependingon the pose of the physical object (O), the display device 100, or anycombination thereof. Additionally, or alternatively, the slice (SL) canchange using any of the input devices described herein. Also, as shownin FIG. 6, the tool 110, 20 can be utilized to change the slice (SL).For example, the tool's 110, 20 pose, position, and/or distance ororientation relative to either the physical object (O) or the displaydevice 100, can automatically cause the control system 60 to change theslice (SL). In some implementations, portions of the slice (SL) can bedisplayed based on a (2D or 3D) virtual boundary (VB) that is associatedwith a tip 114 of the tool 110, 20, as shown in FIG. 4B. For instance,the virtual boundary (VB) can be defined by any shape (e.g., rectangle,box, circle or sphere) having any suitable dimension (e.g., a 200 mmdiameter) with the tool tip 114 at the center of the boundary (VB). Ifthe tool 110, 20 is moved towards the object (O) such that the object(O) intersects the virtual boundary (VB), the control system 60 candynamically present or change the displayed slice (SL) that correspondsto the intersection. The slice (SL) can be presented in whole (as shownin FIG. 4B) or can be cropped (FIG. 6) depending upon the location ofthe virtual boundary (VB).

ii. Contextually Accurate Rendering

With reference to FIG. 7, the control system 60 is configured to renderthe computer images (R) in a contextually accurate manner. In otherwords, the computer image (R) rendering takes into account spatiallayering of the object (O), tool 110, 20, and/or slice (SL) relative toone other in in the common coordinate system so as to mimic the actualspatial location of the same in the real coordinate system in whichthese items exist. As shown in FIG. 7, the tool 110, 20 is physicallyinserted into the object (O) and the display device 100 presents thevirtual model (T) of the tool 110, 20 and the corresponding slice (SL)relative to a 3D model (VM) of the object (O). However, as shown, theserenderings (R) layered based on the perspective of the virtual camera(VC). In other words, the virtual model (T) of the tool 110, 20deliberately obstructed by corresponding portions of the 3D model (VM)of the object (O). In this case, the rib cage of the anatomy isobstructing the tool 110, 20 shaft. Similarly, the slice (SL) isdisplayed such that portions of the slice (SL) are obstructed bycorresponding portions of the 3D model (VM) of the object (O). In thiscase, the slice (SL) is partially obstructed by several ribs. Althoughthis technique obstructs portions of the visualization, it benefits theuser to visualize the environment in a context that closely resemblesreal-world spatial positioning of the object (O), slice (SL) and tool110, 20. This visualization technique can be utilized for any of thetechniques described herein, including the FVF and SPSV techniques, aswell as for any of the specific implementations, conditions, and/orsituations described herein.

iii. Adjustment System

As shown in FIG. 8, the display device 100 may optionally be connectedto an adjustment system 104. The adjustment system 104 can be passive oractive and can comprise any features or possible configurations of therobotic manipulator 14 described above. For example, the adjustmentsystem 104 may comprise a plurality of links (L), joints (J), andactuators (M) for driving the joints (J). The adjustment system 104 canbe manually or automatically controlled to adjust a pose of the displaydevice 100. The pose of the display device 100 can be moved in up tosix-degrees of freedom (three translational and three rotational).Sensors (S) may be provided on any of the components of the adjustmentsystem 104 to detect movement of the adjustment system 104. The controlsystem 60 can utilize measurements from the sensors (S) to kinematicallyderive the pose of the display device 100. This can be done in additionto or alternatively from using the localizer 44. The sensors (S) can beany suitable configuration, including, but not limited to: motor currentsensors, joint position sensors or encoders (rotary, absolute,incremental, virtual-absolute, etc.), optical sensors, inertial sensors(accelerometer, inclinometer, gyroscope, etc.) or the like. Any of thesensors (S) can also be provided on the display device 100 itself.

In some instances, as shown in FIG. 8, one or more user input devices106 can be wired or wirelessly connected to the control system 60 toenable the user to control the pose of the display device 100. The inputdevices 106 include but are not limited to: a foot pedal 106 a, ahand-held pendant 106 b, display input 106 c such as a tablet,smartphone, or display of the navigation system, the tool 110, 20,and/or a viewpoint tracking system (VTS), such as an external camera (C)facing the user or a head-mounted device 120. The input devices can alsobe any of those described above, including but not limited to: pushbuttons, sensors, switches, keyboard, mouse, microphone(voice-activation), gesture control devices, touchscreens, joysticks,etc. The input device 106 receives a command from the user and thecontrol system 60 directs the one or more actuators M to adjust thejoints J and ultimately the pose of the display device 100 in accordancewith the command.

The tracked tool 110, 20 can also be utilized for triggering the controlsystem 60 to control the adjustment system 104. In one implementation,the pose of the tracked tool 110, 20, whether it be position and/ororientation, or any derivate thereof (velocity, acceleration, etc.) cancause the control system 60 to adjust the pose of the display device100. For example, the tracked pose tool 110, 20 can be compared to thevirtual position (VP) of the virtual camera (VC), the viewing frustum(F), the plane (P) of the display device 100, or any combinationthereof. The control system 60 can assess this comparison relative to athreshold condition or measurement. If the control system 60 determinesthat the tool 110, 20 has moved in a manner that meets or exceeds thethreshold condition, the control system 60 can command the adjustmentsystem 104 to adjust the display device 100. This may be beneficial forvarious situations, including, but not limited to: the tool 110, 20moving towards or away from the object (O), the tool 110, 20 movingin/out of the viewing frustum (F), keeping the tool 110, 20 within viewof the virtual camera (VC), or the like.

With continued reference to FIG. 8, the head-mounted device 120 can alsobe utilized for triggering the control system 60 to control theadjustment system 104. In one implementation, the pose of thehead-mounted device 120 is tracked using any tracking techniquedescribed herein, or equivalents thereof. The tracked pose of thehead-mounted device 120, whether it be position and/or orientation, orany derivate thereof (velocity, acceleration, etc.) can cause thecontrol system 60 to adjust the pose of the display device 100. Forexample, the tracked pose of the head-mounted device 120 can be comparedto the virtual position (VP) of the virtual camera (VC), the viewingfrustum (F), the plane (P) of the display device 100, or any combinationthereof. The control system 60 can assess this comparison relative to athreshold condition or measurement. If the control system 60 determinesthat the head-mounted device 120 has moved in a manner that meets orexceeds the threshold condition, the control system 60 can command theadjustment system 104 to adjust the display device 100. This may bebeneficial for various situations, including, but not limited to: thehead-mounted device 120 moving towards or away from the object (O), thehead-mounted device 120 moving in/out of the viewing frustum (F),keeping the head-mounted device 120 within view of the virtual camera(VC), or the like.

Additionally, or alternatively, any other viewpoint tracking system(VTS), such as an external camera (C) facing the user can be utilized totrack the user's viewpoint (V) for triggering the control system 60 tocontrol the adjustment system 104.

The adjustment system 104 can be utilized for any of the techniquesdescribed herein, including the FVF and SPSV techniques, as well as forany of the specific implementations, conditions, and/or situationsdescribed herein.

B. Dynamic Mirror View Frustum Visualization Technique

With reference to FIGS. 9-12, another example visualization method thatcan be utilized with the surgical system 10 comprises a Dynamic MirrorView Frustum (DMVF) for aiding in user interaction with the physicalobject (O).

As will be understood from the description below, there are technicalsimilarities between the FVF and DMVF techniques. Accordingly, any andall of the above description related to the system 10, the FVFtechnique, and any physical, computational, and/or any techniquesassociated therewith are fully incorporated by reference for use withand can be applied by the DMVF technique described herein and hence arenot repeated for simplicity of description.

For the DMVF technique, and with reference to FIG. 9, the display device100 defines the plane (P). The physical object (O) is located in frontof the plane (P). In other words, the physical object (O) is located infront of the displayed screen of the display device 100.

The user viewing the display device 100 is also located in front of theplane (P). The user's viewpoint (V) is tracked utilizing a viewpointtracking system (VTS). In one implementation, the viewpoint trackingsystem (VTS) includes a camera (C) facing the user. The viewpointtracking system (VTS) can the navigation system itself or can be part ofor separate from the navigation system. The camera (C) can be thelocalizer 44 or a separate device. The camera (C) can be mounted to thedisplay device 100, or elsewhere. The control system 60 can receivesignals from the camera (C) and recognize changes to position and/ororientation of the user's face, eyes, head, or other feature using poseestimation algorithms.

In another implementation, the viewpoint tracking system (VTS)additionally or alternatively includes a head-mounted device 120 that isprovided directly on the user. The head-mounted device 120 is located infront of the plane (P) and in front of the displayed screen of thedisplay device 100. In a one configuration, the head-mounted device 120is located on a first side (S1) of the physical object (O) and thedisplay device 100 is located on a second opposing side (S2) of thephysical object.

The head-mounted device 120 comprises one or more trackable features(HT) such that its pose can be tracked by the navigation system 32and/or control system 60. The trackable features (HT) can be any typedescribed above or any equivalents thereof. In this technique, thehead-mounted device 120 is any device that is configured to enabletracking the pose of the user's point of view. Here, pose means positionof the head-mounted device 120, and optionally, position andorientation. The user's point of view can be defined by the generalfield of view of the user's vision, the direction in which the userturns his/her head, and/or the gaze of the user's eyes. The head-mounteddevice 120 can be or include one or more trackers attached to any partof a user's head and/or eye or head wear such as glasses (such as thoseshown in FIG. 8 for example), goggles, head-band helmet, contact lenses,or the like. In one example, the head-mounted device 120 is anoptical-see-through head-mounted-display. Other examples of thehead-mounted device 120 are contemplated. In FIG. 9, the viewpointderived from tracking the head-mounted device 120 is shown as a spherefor simplicity and the viewpoint is facing the display device 100.

Just as with the FVF technique, the navigation system 32 and/or controlsystem 60 registers computer images (R) to the physical object (O). Thenavigation system 32 and/or control system 60 track a pose of thephysical object (O), the display device 100, and the user's viewpoint(V) relative to one another in a common coordinate system. The displaydevice 100 is controlled for rendering the registered computer images(R) according to the tracked pose of the physical object (O), thedisplay device 100, and the user's viewpoint (V). In other words, thepose of each of these items can affect how the computer images (R) aredisplayed.

A perspective of the computer image (R) is based on a field of view orviewing frustum (F) of a virtual camera (VC) that has a virtual position(VP) behind the plane (P), shown as side (S3) in FIG. 9, which is behindthe display device 100. The virtual camera (VC) faces the rear of thedisplay device 100 and faces the viewpoint of the user as derived fromthe viewpoint tracking system (VTS). The physical object (O), thedisplay device 100, and the user's viewpoint (V) are at least partiallywithin the viewing frustum (F) in FIG. 9. Since the virtual camera (VC)is located behind the plane (P) of the display device 100, the rendering(R) provides visualization as though the display device 100 were amirror relative to the viewpoint of the user. In FIG. 9, the computerimage (R) is showing the side of the physical object (O) at side (S2).

The virtual position (VP) of the virtual camera (VC) is automaticallyupdated in response to adjustment to the tracked pose of the user'sviewpoint (V). The user can move left and right or back and forth andsee the physical object (O) and any tool 110, 20 moving correspondinglyon the mirror-like rendering (R) of the display device 100. By knowingthe poses of the physical object (O), the display device 100, and theuser's viewpoint (V), the control system 60 is able to create thecomputer images (R) such that they obey the same laws as a real mirror.

FIG. 10A-10F illustrate examples of the DMVF technique. Specifically,simulated mirror views are shown in FIGS. 10A-10D and real-worldillustrative examples are shown in FIGS. 10E and 10F. Comparing thethird person views, as shown in FIGS. 10A and 10C, the virtual position(VP) of the virtual camera (VC) changes depending on user's viewpoint(V) (shown as a sphere) as derived from viewpoint tracking system (VTS).FIG. 10B shows the first-person viewpoint of what would be seen on thedisplay device 100 in the scenario of FIG. 10A. FIG. 10C shows thefirst-person viewpoint of what would be seen on the display device 100in the scenario of FIG. 10D.

The display device 100 shows the objects in front of the display justlike a mirror would do, taking the poses of the screen, objects, and theuser viewpoint into account. This paradigm works can contribute to amore intuitive visualization of the physical object (O) by using motionparallax and observing structures from additional desired viewpoints.The user gains the freedom to interact with the data just by looking,without necessarily requiring an interaction device like a mouse orjoystick. As the users do not constantly need to redefine their view onthe object (O), the user is enabled to toggle the interactivity on oroff for example with any input device, such as a foot pedal. The usercan change the visualization in a hands-free manner. Since the naturalhuman visual system is adapted to looking at real mirrors, the DMVFvisualization provides an intuitive interface. DMVF also facilitatesexploring and defining the best slice (SL) or rendering (R) for a givennavigation task. The slice (SL) or rendering (R) can remain fixed duringthe surgical action until further data exploration is desired.

i. Virtual Camera and Mirror Frustum Setup for DMVF

The virtual position (VP) of the virtual camera (VC) dynamically changesin the DMVF method depending upon at least the tracked pose of theuser's viewpoint (V). In one implementation, the viewpoint of thevirtual camera (VC) can be derived as described herein. In one example,the control system 60 moves the virtual camera (VC) according to aprojection matrix that is adjusted so that the viewing frustum (F) isfitted such that boundary features (BF) of the frustum (F) coincide withthe features (DF) of the display device 100. The features (DF) of thedisplay device 100 comprise points or other geometry which encode asize, aspect ratio, position and orientation of the display device 100in the common or world coordinate system. In the example of FIG. 9,wherein the viewing frustum (F) comprises a truncated four-sided pyramidand the display device 100 is rectangular, the boundary features (BF) ofthe viewing frustum (F) are points on the edges of the sides of thepyramid and the features (DF) are the fixed corners of the displaydevice 100. Hence, the points on the edges of the viewing frustum (F)are made to coincide with the corners of the display device 100 for anypositional movement of the virtual camera (VC) in the coordinate system.Of course, this configuration may change depending on the geometry ofthe viewing frustum (F) and the display device 100. This results in awarped image (R), which appears in correct perspective as a mirrorreflection from the tracked pose of the user's viewpoint (V).

From the virtual position (VP) of the virtual camera (VC), a mirroredworld position of the viewpoint p_(mirrored) can be calculated asfollows: define the normal to the mirror plane (P) as (0; 0; 1). In itslocal coordinate system, the mirroring corresponds to multiplicationwith the matrix Mf, which is expressed in [1] and which considers thefixed features (DF) of the display device 100 (i.e., in this case thefour corners).

$\begin{matrix}{M_{flip} = {\begin{matrix}1 & 0 & 0 & 0 \\0 & 1 & 0 & 0 \\0 & 0 & {- 1} & 0 \\0 & 0 & 0 & 1\end{matrix}}} & \lbrack 1\rbrack\end{matrix}$

This is equivalent to scaling by −1 in z-direction. Let us assume,according to one implementation, that to provide the mirror view, therendering depends on the position, and not the orientation, of theuser's viewpoint (V) with respect to the mirror plane (P). A firsttransform is performed to transform a world position of the virtualcamera (VC) viewpoint p into local coordinates of the mirror. Thiscauses rotation of the frustum (F) to the viewpoint of the user. Thenmirroring is performed with matrix M_(flip) which provides perspectiveprojection relative to the display device 100. Lastly, the mirroredlocal coordinates of the mirror are transformed back to the worldposition of the virtual camera (VC) viewpoint p which translates theuser's viewpoint (V) to the apex of the viewing frustum (F). The resultis a projection matrix expressed by:

p _(mirrored)=Mirror_(T) _(World) ·M _(flip)·World_(T) _(Mirror·p)   [2]

In the equation expressed in [2], the notation A_(T) _(B) is utilized torepresent a transformation from coordinate system A into B. Thiscomputation enables re-alignment or rotation of the base (B) orfar-clipping plane (FCP) of the frustum (F) from an otherwise XYcoordinate aligned position relative to the display device 100 to analignment that is positioned relative to the user's viewpoint (V).Thereafter, the user viewpoint-aligned base (B) or far-clipping plane(FCP) is re-aligned back to the XY coordinates of the display device 100such that an off-axis projection matrix can be applied to applyperspective rendering of the computer images (R).

To enable the computer images (R) on the display device 100 to rendermirror views of the physical object (O) or tools 110, 20 in front of thedisplay device 100 in relation to the user's pose, the off-axisprojection matrix can be employed in one implementation. Here off-axismeans that a line drawn from the user's viewpoint (V) to the displaydevice 100 would not be at the geometrical center of the display device100, but rather off-center. In such situations, the viewing frustum (F)becomes asymmetric and will change shape as the user's viewpoint (V)changes. The location at which the user's viewpoint (V) relative to thegeometry of the plane (P) of the display device 100 can be monitored bythe control system 60. As the frustum (F) shape changes depending uponthe user's viewpoint (V), the boundary features (BF) of the viewingfrustum (F) change. In one implementation, the viewing frustum (F) canbe recomputed and/or updated at frequent intervals, such as, e.g., atevery frame, as the pose of the display device 100 and/or the viewpointof the user are likely to change.

The off-axis projection matrix has the mirrored viewpoint p at the apexof the viewing frustum (F) and the base (B) matching the surface orplane (P) of the display device 100. The viewing frustum (F) is rotatedto align with the user's viewpoint (V). This takes the position of theviewpoint into account and warps the image so that it appears like aprojection onto a tilted plane relative to the plane (P) of the displaydevice 100. The effect of the changed viewpoint of the user on thefrustum (F) together with the warped images, can be seen in FIGS. 10A,10B as compared with FIGS. 10C, 10D. This implementation relies on agraphic API to set the projection matrix with the frustum (F) defined bythe near clipping plane (NCP) coordinates in view space, which is thenrotated to be non-perpendicular and moved to be located at the mirroredviewpoint. In other words, the base (B) of the frustum (F) isdynamically modified rotate out of the X-Y plane of the plane (P) of thedisplay device 100 and positioned corresponding to the angle of theuser's viewpoint (V) as derived from the viewpoint tracking system(VTS).

The calculation provided above provides one implementation of virtualcamera (VC) setup for DMVF. However, there may be alternative manners ofdynamically changing the virtual position (VP) of the virtual camera(VC) depending upon at least the user's viewpoint (V), without departingfrom the scope of the concept. For example, it is possible to align thebase (B) of the frustum (F) to the display device 100. Boundary features(BF) of the viewing frustum (F) may exceed the geometry of the displaydevice 100. In such instances, the control system 60 may limit theboundary features (BF) of the viewing frustum (F) relative to thegeometry of the display device 100.

FIGS. 11A-11C compare, in various views of the X,Y,Z coordinate system,the relative positioning between the virtual camera (VC) and frustum(F), user's viewpoint (V), display device 100 and object (O) among threeexample scenarios. In these examples, the position of the virtual camera(VC) and the user's viewpoint (V) can be mirrored relative to the plane(P) of the display device 100 for each of the X, Y, and Z axesindividually. For instance, the virtual camera (VC) and the user'sviewpoint (V) are substantially equal distance apart from the displaydevice 100 in each of the X, Y, and Z-directions. When visualized fromthe X-Y plane, the position of the virtual camera (VC) and the user'sviewpoint (V) appear to coincide as result of this mirroring effect.However, perfect mirroring is not required in all implementations andthe distance of (VC) and (V) to the display device 100 may not be equalfor each of the axes and such distances can be customized or changed.

In FIG. 11A, the user's viewpoint (V) is on-axis at given positionrelative to the display device 100. The base (B) of the frustum (F) ofvirtual camera (VC) aligns with the X-Y plane of the display device 100.The projection renders computer images (R) that are centered and alignedto the user's centered viewpoint.

In FIG. 11B, the user's viewpoint (V) is moved to the left of center(from the user's perspective) and moved away from the display device 100further than the given position of FIG. 11A. This relative change inmotion of the user's viewpoint (V) causes the virtual camera (VC) tochange correspondingly. In other words, the virtual camera (VC) moves tothe right and away from the display device 100 (from the virtualcamera's perspective). This correctly positions the virtual camera (VC)relative to the user's viewpoint (V). The viewing frustum (F) is rotated(counter-clockwise as seen from above) out of the X-Y plane of thedisplay device 100 to the align to the user's viewpoint (V). In someinstances, modification of the frustum (F) causes the far-clipping plane(FCP) to move further from the apex of the frustum (F) in response tothe position of the virtual camera (VC) moving away from the displaydevice 100. The projection renders computer images (R) that are rotatedto align to the user's off-center viewpoint. The computer images (R)also render objects, such as the physical object (O) within the frustum(F) to be smaller in size as compared to FIG. 11A since the position ofthe user's viewpoint (V) is further away from the display device 100.

In FIG. 11C, the user's viewpoint (V) is moved to the right (from theuser's perspective) and moved closer to the display device 100 than thegiven position of FIG. 11A. This relative change in motion of the user'sviewpoint (V) causes the virtual camera (VC) to move to the left andcloser to the display device 100 (from the virtual camera'sperspective). This correctly positions the virtual camera (VC) relativeto the user's viewpoint (V). The viewing frustum (F) is rotated(clockwise as seen from above) out of the X-Y plane of the displaydevice 100 to the align to the user's viewpoint (V). In some instances,modification of the frustum (F) causes the far-clipping plane (FCP) tomove closer to the apex of the frustum (F) in response to the positionof the virtual camera (VC) moving away from the display device 100. Theprojection renders computer images (R) that are rotated to align to theuser's off-center viewpoint. The computer images (R) also renderobjects, such as the physical object (O) within the frustum (F) to belarger in size as compared to FIG. 11A since the position of the user'sviewpoint is closer to the display device 100.

The dimensioning of the components of the FIG. 11 are merely providedfor illustrative purposes and may not be exactly to scale. Furthermore,although the X-Z plane is shown illustrating lateral motion among theuser's viewpoint and virtual camera (VC), the principles describedherein may apply fully to vertical motion between the same in the X-Yplane.

ii. Viewpoint Facing Slice Visualization

Referring now to FIGS. 12A and 12B, a Viewpoint Facing SliceVisualization (VFSV) sub-technique of the DMVF method is described.While the DMVF visualization is intuitive to use and can display 3Ddata, it is contemplated to further incorporate the slice view into thespatially-aware DMVF visualization concept described.

The slicing of the data can occur according to any of the techniquesdescribed above with respect to the SPSV technique and as shown withrespect to FIGS. 4A and 4B, and hence, are not repeated herein forsimplicity. Furthermore, the selection of the displayed slice (SL) canoccur according to any of the techniques described above with respect tothe SPSV technique and as shown with respect to FIGS. 4A and 4B.

In one implementation, the tool 110, 20 can be utilized to change theslice (SL). For example, the tool's 110, 20 pose, position, and/ordistance or orientation relative to either the physical object (O) orthe display device 100, can automatically cause the control system 60 tochange the slice (SL). In some implementations, portions of the slice(SL) can be displayed based on a (2D or 3D) virtual boundary (VB) thatis associated with a tip 114 of the tool 110, 20, as shown in FIG. 4B.For instance, the virtual boundary (VB) can be defined by any shape(e.g., rectangle, box, circle or sphere) having any suitable dimension(e.g., a 200 mm diameter) with the tool tip 114 at the center of theboundary (VB). If the tool 110, 20 is moved towards the object (O) suchthat the object (O) intersects the virtual boundary (VB), the controlsystem 60 can dynamically present or change the displayed slice (SL)that corresponds to the intersection. The slice (SL) can be presented inwhole (as shown in FIG. 4B) or can be cropped (FIG. 6) depending uponthe location of the virtual boundary (VB).

Additionally, or alternatively, the slice (SL) may be changed dependingon the pose of the physical object (O), the display device 100, user'sviewpoint or any combination thereof. The slice (SL) may be changedusing any of the input devices 106 described herein.

The orientation of the slice (SL) can be manipulated according toseveral different techniques. In the VFSV technique, the control system60 tracks the display device 100 and the user's viewpoint (V) accordingto the DMVF technique. A position of the slice (SL) is based on thetracked position of the tool 110, 20. The orientation of the slice (SL),however, is chosen to point towards the mirror viewpoint. Thisguarantees that the slice (SL) faces the user when viewed through themirror. That is, the slice (SL) is oriented to face the user's viewpoint(V) as derived by the viewpoint tracking system (VTS). The orientationof the slice (SL) rendered using the DMVF technique described above.That is, to accurately account for the user's viewpoint (V) relative tothe display device 100, the slice (SL) is rotated relative to the plane(P) of the display device 100 and projected to be smaller or larger.

An example of the VFSV technique from two different first-personviewpoints can be seen in FIGS. 12A and 12B, wherein the tool 110, 20remains stationary between FIGS. 12A and 12B. In FIG. 12B, the user'sviewpoint (V) is moved slightly to the left, and closer to the tool 110,20. The distance of the user's viewpoint (V) to the display device 100remains about the same. In this example, the displayed slice (SL)appears to be the same to the user since the slice (SL) has rotated tocorrespond to the lateral motion of the user's viewpoint (V). If theuser in FIG. 12B were to approach the display device 100, the slice (SL)will appear correspondingly larger, and vice-versa.

While the user's viewpoint (V) is utilized to implement the VFSVtechnique, the orientation of the computer image (R) of the slice (SL)can additionally be adjusted based on the pose of the display device100, the pose of the object (O) and/or the pose of the tracked tool 110,20 brought within the viewing frustum (F) of the virtual camera (VC).

Additionally, the control system 60 is configured to render the computerimages (R) in a contextually accurate manner for the VFSV technique. Inother words, the computer image (R) rendering takes into account spatiallayering of the object (O), tool 110, 20, and/or slice (SL) relative toone other in in the common coordinate system so as to mimic the actualspatial location of the same in the real coordinate system in whichthese items exist. Hence, any of the description above related to thecontextually accurate rendering for the SPSV technique, including thevisualization of FIGS. 6 and 7 can be applied in its entirety to theVFSV technique and is not repeated for simplicity in description.

C. Visualization Mode Switching

The control system 60 can enable switching between any of thevisualization modes described herein at any time and in response to anyinput or condition. For example, the control system 60 can enableswitching between any of the following: FVF and DMVF visualizations;SPSV and the VFSV visualizations; 3D model visualization (withoutslices) using FVF visualization and using the SPSV technique (with orwithout the 3D model); 3D model visualization (without slices) using theDMVF technique and visualization using the VFSV technique (with orwithout the 3D model); static visualization and any of the FVF, SPSV,DMVF and VFSV techniques.

The switching between visualization modes can occur depending on userinput. For example, the control system 60 can receive a command from anyof the input devices 106 described herein, which include but are notlimited to: a foot pedal 106 a, a hand-held pendant 106 b, display input106 c such as a tablet, smartphone, display device 100 or display of thenavigation system, the tool 110, 20, and/or a head-mounted device 120,push buttons, sensors, switches, keyboard, mouse, microphone(voice-activation), gesture control devices, touchscreens, joysticks,etc.

The switching between visualization modes can occur automaticallydepending upon conditions such as, but not limited to: a type ofsurgical procedure, a certain step of the procedure, a tracked pose ofthe user's viewpoint (V) or head-mounted device 120, a tracked pose ofthe object (O), a tracked pose of the display device 100, a tracked poseof a tool 110, 20, or any combination thereof.

Several implementations have been discussed in the foregoingdescription. However, the implementations discussed herein are notintended to be exhaustive or limit the invention to any particular form.The terminology which has been used is intended to be in the nature ofwords of description rather than of limitation. Many modifications andvariations are possible in light of the above teachings and theinvention may be practiced otherwise than as specifically described.

D. Case Study and Experimental Results

As proof of the concept, a confidential case study was performed withthree trauma surgeons (chief surgeon, attending physician, andresident). The set up included a 3D-printed patient phantom, a displaydevice 100 on an adjustable desk mount fixed to the OR table, and aStryker® Flashpoint 6000 tracking system to track the patient phantom,the tool 110, 20, the display device 100, and the viewpoint. The3D-printed patient phantom contained segmented anatomy and acorresponding CT volume. During the experiments, surgeons explored theproposed techniques in a predefined order with a think-aloud protocoland no time limit, followed by a semi-structured interview. The surgeonswere asked to pay attention to differences in the visualization methodsand to consider possible application scenarios. First, surgeons wereexposed to use the conventional orthogonal slice visualizationcontrolled by a tracked instrument. Next, the surgeons were presentedthe novel FVF and SPSV visualization techniques. Thereafter, thesurgeons explored the methods with head tracking, namely, the DMVF andVFSV visualization techniques. When using FVF and DMVF, the surgeonswere instructed to switch between two modes: segmented 3D structures anddirect volume rendering. In the interviews, the surgeons praised theidea to automate slice presentation and to present the images to themthat match the orientation of their direct view onto the patient. Theoverall feedback of the experts on the proposed concept was extremelypositive. The surgeons appreciated the fact that for DMVF and VFSV, thesurgeons could interact with patient data in a sterile manner, whichthey considered to be important when an intervention turns out to bemore complex than initially expected. The participants agreed that theFVF and SPSV methods could be easily integrated into the establishedsurgical workflow. When asked about possible applications of theproposed concept, the answers included use-cases in complexcraniomaxiofacial surgery, and interventions in orthopedics and traumasurgery, and interventions where high precision is utilized, such asendoprothetics. Finally, the participants agreed that the proposedvisualization techniques can help to familiarize with the specificpatient anatomy and understand how instruments are currently positionedwith respect to a planned trajectory. Participants unanimously agreethat the new visualization techniques help with orientation with respectto obtaining an overview of the specific patient anatomy and how theinstruments are currently positioned.

What is claimed is:
 1. A system for aiding in interaction with aphysical object, the system comprising: a display device defining aplane and being located on a first side of the physical object; and anavigation system coupled to a control system and being configured to:register computer images to the physical object; track a pose of thephysical object and the display device in a common coordinate system;and control the display device to render the registered computer imagesaccording to the tracked pose of the physical object and the displaydevice; and wherein a perspective of the rendering is based on a virtualcamera that has a virtual position located on a second side of thephysical object opposite to the first side, and wherein the virtualcamera has a field-of-view that faces the plane of display device, andwherein the virtual position of the virtual camera updates automaticallyin response to adjustment to the pose of the display device.
 2. Thesystem of claim 1, wherein the computer images of the physical objectare derived from a 3D model, and the control system is configured tocontrol the display device to display one or more slices of the 3D modeldepending upon the tracked pose of the physical object and the displaydevice.
 3. The system of claim 2, wherein the one or more slices aresliced at planes parallel to the plane of the display device.
 4. Thesystem of claim 2, wherein the control system is configured to controlthe display device to automatically change the one or more slices toother slices in response to one or more of the following: the trackedpose of the display device; the tracked pose of the physical object; atracked pose of a surgical instrument; and a tracked pose of a user'sviewpoint.
 5. The system of claim 1, wherein the virtual position of thevirtual camera is located at a predetermined distance from the plane ofdisplay device and wherein the predetermined distance updatesautomatically in response to at least one or more of the following: thetracked pose of the display device, the tracked pose of the physicalobject, a tracked pose of a surgical tool; a tracked pose of a user'sviewpoint; a type of surgical procedure; and a certain step of asurgical procedure.
 6. The system of claim 1, wherein the virtualposition of the virtual camera is located at a predetermined distancefrom the plane of display device, and wherein the predetermined distanceis fixed.
 7. The system of claim 1, wherein an X-Y placement of thevirtual position of the virtual camera relative to the plane of thedisplay device updates automatically in response to at least one or moreof the following: the tracked pose of the display device, the trackedpose of the physical object, a tracked pose of a surgical tool; atracked pose of a user's viewpoint; a type of surgical procedure; and acertain step of a surgical procedure.
 8. The system of claim 1, whereinan X-Y placement of the virtual position of the virtual camera relativeto the plane of the display device is fixed.
 9. The system of claim 1,wherein the pose of the display device is manually adjustable, and thevirtual position of the virtual camera updates automatically in responseto manual adjustment to the pose of the display device.
 10. The systemof claim 1, comprising one or more actuators coupled to the displaydevice and the control system being configured to control the one ormore actuators to adjust the pose of the display device.
 11. The systemof claim 10, comprising an input device coupled to the control systemand the control system is configured to receive a command from the inputdevice and control the one or more actuators to adjust the pose of thedisplay device in accordance with the command.
 12. The system of claim10, comprising a surgical instrument including one or more trackablefeatures, and wherein the navigation system is configured to track apose of the surgical instrument in the common coordinate system andcontrol the display device to display an image of the surgicalinstrument, and wherein the control system is configured to control theone or more actuators to adjust the pose of the display device based onthe tracked pose of the surgical instrument.
 13. The system of claim 10,comprising a viewpoint tracking system coupled to the navigation systemand being configured to track a pose user's viewpoint in the commoncoordinate system and wherein the control system is configured tocontrol the one or more actuators to adjust the pose of the displaydevice based on the tracked pose related to the user's viewpoint.
 14. Amethod of operating a system for aiding in interaction with a physicalobject, the system comprising a display device defining a plane andbeing located on a first side of the physical object, and a navigationsystem coupled to a control system, the method comprising: registeringcomputer images to the physical object; tracking a pose of the physicalobject and the display device in a common coordinate system; andcontrolling the display device to render the registered computer imagesaccording to the tracked pose of the physical object and the displaydevice; and wherein a perspective of the rendering is based on a virtualcamera having a virtual position located on a second side of thephysical object opposite to the first side, and wherein the virtualcamera has a field-of-view facing the plane of display device, andautomatically updating the virtual position of the virtual camera inresponse to adjusting the pose of the display device.
 15. A system foraiding in interaction with a physical object, the system comprising: adisplay device defining a plane, wherein the physical object is locatedin front of the plane; a navigation system coupled to a control systemand being configured to: register computer images to the physicalobject; track a pose of the physical object, the display device, and auser's viewpoint in a common coordinate system; and control the displaydevice for rendering the registered computer images according to thetracked pose of the physical object, the display device, and the user'sviewpoint; and wherein a perspective of the rendering is based on afield-of-view of a virtual camera that has a virtual position behind theplane, and wherein the virtual position of the virtual camera isautomatically updated in response to movement of the tracked pose of theuser's viewpoint.
 16. The system of claim 15, wherein the field-of-viewof the virtual camera is automatically updated in response to movementof the tracked pose of the user's viewpoint.
 17. The system of claim 16,wherein the display device comprises fixed features and wherein thefield-of-view of the virtual camera comprises boundary features andwherein the control system is configured to fit the boundary features tocoincide with the fixed features for any given virtual position of thevirtual camera that is automatically updated in response to movement ofthe tracked pose of the user's viewpoint.
 18. The system of claim 15,wherein the perspective of the rendering is automatically rotated toalign with the tracked pose of the user's viewpoint relative to theplane of the display device.
 19. The system of claim 15, wherein thevirtual position of the virtual camera is automatically updated to: movetowards the plane of the display device in response to movement of thetracked pose of the user's viewpoint towards the plane of the displaydevice; and move away from the plane of the display device in responseto movement of the tracked pose of the user's viewpoint away the planeof the display device.
 20. The system of claim 15, wherein the controlsystem renders the computer images to: increase in size in response tomovement of the tracked pose of the user's viewpoint towards the planeof the display device; and decrease in size in response to movement ofthe tracked pose of the user's viewpoint away from the plane of thedisplay device.
 21. The system of claim 15, wherein the computer imagesof the physical object are derived from a 3D model, and the controlsystem is configured to control the display device to display one ormore slices of the 3D model depending upon the tracked pose of thephysical object, the display device, and the user's viewpoint.
 22. Thesystem of claim 21, wherein a perspective of the one or more slices isrendered based on the field-of-view of the virtual camera that has thevirtual position automatically updated in response to movement of thetracked pose of the user's viewpoint.
 23. The system of claim 21,wherein the control system is configured to control the display deviceto automatically change the one or more slices to other slices inresponse to one or more of the following: the tracked pose of the user'sviewpoint; the tracked pose of the display device; the tracked pose ofthe physical object; and a tracked pose of a surgical instrument. 24.The system of claim 15, wherein the navigation system comprises ahead-mounted device and wherein the navigation system is configured totrack the pose of the user's viewpoint by being configured to track apose of the head-mounted device.
 25. The system of claim 15, wherein thenavigation system comprises a camera configured to be directed towardsthe user and wherein the navigation system is configured to track thepose of the user's viewpoint by being configured to track a pose of theuser's head, face, or eyes using the camera.
 26. The system of claim 25,wherein the camera is mounted to the display device.
 27. A method ofoperating a system for aiding in interaction with a physical object, thesystem comprising a display device defining a plane, wherein thephysical object is located in front of the plane, and a navigationsystem coupled to a control system, the method comprising: registeringcomputer images to the physical object; tracking a pose of the physicalobject, the display device, and a user's viewpoint in a commoncoordinate system; and controlling the display device for rendering theregistered computer images according to the tracked pose of the physicalobject, the display device, and the user's viewpoint; and wherein aperspective of the rendering is based on a field-of-view of a virtualcamera having a virtual position behind the plane, and automaticallyupdating the virtual position of the virtual camera in response tomovement of the tracked pose of the user's viewpoint.